The invention is related to the field of memory processors, and in particular to exporting computational capabilities into a block-oriented disk memory.
A key component of most modern computing systems is a disk—a device that can store large amounts of data in a compact format, but that can access such data only in relatively large blocks. Current devices fall in at least two categories: (1) rotating hard drives where a magnetic medium is spun underneath a read/write head, and (2) a solid state drives (SSD) constructed primarily from “flash” chips that store data as charge in semi-conductor devices. In both cases, access is on a “block” basis: a read or write request must handle some integral number of blocks, each of a size of a few hundred to a few thousand bits at a time.
For any data access to the entirety of a large data object such as a file that might scale into the gigabyte range, this blocking is at best just an inconvenience. However, any request that wishes to access only a few bits to different “random” locations in the object must still access a minimum of a whole block, and if those bits must be modified, the modification must be done to a copy of the block outside of the device, and then sent back in. For example, to modify a 32 bit entity within a 4096 bit block the device must read out 4096 bits and then accept 4096 bits in return. The time to transfer this 8192 bits over the interface to the disk is 256 time longer than to transfer the 32 bits that have to change.
Even worse, if different parts of the same block may in any way be subject to simultaneous updates from several different processes external to the disk, every single one of these updates must perform this double block transfer, with some additional (and perhaps quite complex) locking and/or synchronization to prevent some of the updates from being lost.
The net result is that many non-sequential accesses to small amounts of data can have an extraordinarily deleterious effect on the disk's performance. As we move towards big data problems where such non-uniform accesses become more and more common, the inability to do anything other than read and write relatively large blocks of data is rapidly becoming the overall bottleneck to increasing performance, regardless of how much external processing is available.